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Abstract 



This paper examines the problem of ranking a collection of objects using pairwise 
comparisons (rankings of two objects). In general, the ranking of n objects can be 
identified by standard sorting methods using n log 2 n pairwise comparisons. We 
are interested in natural situations in which relationships among the objects may 
allow for ranking using far fewer pairwise comparisons. Specifically, we assume 
that the objects can be embedded into a d-dimensional Euclidean space and that 
the rankings reflect their relative distances from a common reference point in R d . 
We show that under this assumption the number of possible rankings grows like 
n 2d and demonstrate an algorithm that can identify a randomly selected ranking 
using just slightly more than dlogn adaptively selected pairwise comparisons, 
on average. If instead the comparisons are chosen at random, then almost all 
pairwise comparisons must be made in order to identify any ranking. In addition, 
we propose a robust, error-tolerant algorithm that only requires that the pairwise 
comparisons are probably correct. Experimental studies with synthetic and real 
datasets support the conclusions of our theoretical analysis. 

1 Introduction 

This paper addresses the problem of ranking a set of objects based on a limited number of pair- 
wise comparisons (rankings between pairs of the objects). A ranking over a set of n objects 
O = 62, ■ ■ ■ , n ) is a mapping <j : {1, . . . , n) —¥ {1, . . . , n) that prescribes an order 



where 6^ -< 6j means 6{ precedes 9j in the ranking. A ranking uniquely determines the collection 
of pairwise comparisons between all pairs of objects. The primary objective here is to bound the 
number of pairwise comparisons needed to correctly determine the ranking when the objects (and 
hence rankings) satisfy certain known structural constraints. Specifically, we suppose that the objects 
may be embedded into a low-dimensional Euclidean space such that the ranking is consistent with 
distances in the space. We wish to exploit such structure in order to discover the ranking using a 
very small number of pairwise comparisons. To the best of our knowledge, this is a previously open 
and unsolved problem. 

There are practical and theoretical motivations for restricting our attention to pairwise rankings that 
are discussed in Section|2] We begin by assuming that every pairwise comparison is consistent with 
an unknown ranking. Each pairwise comparison can be viewed as a query: is 9i before Qfi Each 
query provides 1 bit of information about the underlying ranking. Since the number of rankings is 
n!, in general, specifying a ranking requires <d(n log n) bits of information. This implies that at least 
this many pairwise comparisons are required without additional assumptions about the ranking. In 
fact, this lower bound can be achieved with a standard adaptive sorting algorithm like binary sort 
|TJ. In large-scale problems or when humans are queried for pairwise comparisons, obtaining this 
many pairwise comparisons may be impractical and therefore we consider situations in which the 
space of rankings is structured and thereby less complex. 



a(Q) := -< 6> CT ( 2 ) -<•••-< ^(n-l) ~< #cr(n) 



(1) 
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A natural way to induce a structure on the space of rankings is to suppose that the objects can be 
embedded into a <i-dimensional Euclidean space so that the distances between objects are consistent 
with the ranking. This may be a reasonable assumption in many applications, and for instance the 
audio dataset used in our experiments is believed to have a 2 or 3 dimensional embedding (2) . We 
further discuss motivations for this assumption in Section|2] It is not difficult to show (see Section[3]) 
that the number of full rankings that could arise from n objects embedded in W l grows like n 2d , and 
so specifying a ranking from this class requires only 0(dlog n) bits. The main results of the paper 
show that under this assumption a randomly selected ranking can be determined using 0(d\ogn) 
pairwise comparisons selected in an adaptive and sequential fashion, but almost all (™) pairwise 
rankings are needed if they are picked randomly rather than selectively. In other words, actively 
selecting the most informative queries has a tremendous impact on the complexity of learning the 
correct ranking. 



1.1 Problem statement 



Let tr denote the ranking to be learned. The objective is to learn the ranking by querying the reference 
for pairwise comparisons of the form 

<!,, ■= {<>, -< 9j}. (2) 

The response or label of q^j is binary and denoted as yij := l{q L .j} where 1 is the indicator 
function; ties are not allowed. The main results quantify the minimum number of queries or labels 
required to determine the reference's ranking, and they are based on two key assumptions. 

Al Embedding: The set of n objects are embedded in R d (in general position) and we will also use 
6\, . . . , 9 n to refer to their (known) locations in Mr, Every ranking a can be specified by a reference 
point r a € R d , as follows. The Euclidean distances between the reference and objects are consistent 
with the ranking in the following sense: if the a ranks Oi -< Oj, then — rv|| < \\6j — r a \\. Let 
E n> d denote the set of all possible rankings of the n objects that satisfy this embedding condition. 

The interpretation of this assumption is that we know how the objects are related (in the embedding), 
which limits the space of possible rankings. The ranking to be learned, specified by the reference 
(e.g., preferences of a human subject), is unknown. Many have studied the problem of finding 
an embedding of objects from data ||3] |4] |5). This is not the focus here, but it could certainly 
play a supporting role in our methodology (e.g., the embedding could be determined from known 
similarities between the n objects, as is done in our experiments with the audio dataset). We assume 
the embedding is given and our interest is minimizing the number of queries needed to learn the 
ranking, and for this we require a second assumption. 

A2 Consistency: Every pairwise comparison is consistent with the ranking to be learned. That is, if 
the reference ranks Oi -< Oj, then Oi must precede Oj in the (full) ranking. 



As we will discuss later in Section 3.2 these two assumptions alone are not enough to rule out 
pathological arrangements of objects in the embedding for which at least il(n) queries must be 
made to recover the ranking. However, because such situations are not representative of what is 
typically encountered, we analyze the problem in the framework of the average-case analysis [6|. 

Definition 1. With each ranking a £ E„ j( j we associate a probability n a such that X^es d n a = 
1. Let 7r denote these probabilities and write a ~ n for shorthand. The uniform distribution 
corresponds to ir a — for all a £ En,d> an d we write a ~ IA for this special case. 

Definition 2. If M n (a) denotes the number of pairwise comparisons requested by an algorithm to 
identify the ranking a, then the average query complexity with respect to it is denoted by E ff 

The main results are proven for the special case of tt = U, the uniform distribution, to make the 
analysis more transparent and intuitive. However the results can easily be extended to general dis- 
tributions 7r that satisfy certain mild conditions (see Appendix |A.5| l. All results henceforth, unless 
otherwise noted, will be given in terms of (uniform) average query complexity and we will say such 
results hold "on average." 

Our main results can be summarized as follows. If the queries are chosen deterministically or ran- 
domly in advance of collecting the corresponding pairwise comparisons, then we show that almost 
all (j) pairwise comparisons queries are needed to identify a ranking under the assumptions above. 
However, if the queries are selected in an adaptive and sequential fashion according to the algorithm 
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Query Selection Algorithm 



input: n objects in M. d 
initialize: objects 6 = {9i, .... 9 n } in uni- 
formly random order 

for j=2,. . . ,n 
for i=l,. . . j-l 
if Qij is ambiguous, 

request qtj's label from reference; 
else 

impute <7i,/s label from previously 
labeled queries. 

output: ranking of n objects 





<?2,3 


'02 1/ 


\ '03 



Figure 1: Sequential algorithm for selecting 
queries. See Figure [2] and Section 4.2 for the 
definition of an ambiguous query. 



Figure 2: Objects 0\, #2,6*3 and queries. The 
r a lies in the shaded region (consistent with the 
labels of 91,2,91,3,92.3)- The dotted (dashed) 
lines represent new queries whose labels are 
(are not) ambiguous given those labels. 



in Figure [T| then we show that the number of pairwise rankings required to identify a ranking is no 
more than a constant multiple of d log n, on average. The algorithm requests a query if and only 
if the corresponding pairwise ranking is ambiguous (see Section |4.2) , meaning that it cannot be 
determined from previously collected pairwise comparisons and the locations of the objects in Mr. 
The efficiency of the algorithm is due to the fact that most of the queries are unambiguous when 
considered in a sequential fashion. For this very same reason, picking queries in a non-adaptive or 
random fashion is very inefficient. It is also noteworthy that the algorithm is also computati onally 



efficient with an overall complexity no greater than 0(n poly(rf) poly(logn)) (see Appendix A.l 
In Section|5]we present a robust version of the algorithm of Figure [TJ that is tolerant to a fraction of 
errors in the pairwise comparison queries. In the case of persistent errors (see Section [5]l we show 
that we can find a probably approximately correct ranking by requesting just 0(dlog 2 n) pairwise 
comparisons. This allows us to handle situations in which either or both of the assumptions, Al and 
A2, are reasonable approximations to the situation at hand, but do not hold strictly (which is the case 
in our experiments with the audio dataset). 

Proving the main results involves an uncommon marriage of ideas from the ranking and statistical 
learning literatures. Geometrical interpretations of our problem derive from the seminal works of 
Q in ranking and O in learning. From this perspective our problem bears a strong resemblance 
to the halfspace learning problem, with two crucial distinctions. In the ranking problem, the under- 
lying halfspaces are not in general position and have strong dependencies with each other. These 
dependencies invalidate many of the typical analyses of such problems |0[TO). One popular method 
of analysis in exact learning involves the use of something called the extended teaching dimension 
IfTTI . However, because of the possible pathological situations alluded to earlier, it is easy to show 
that the extended teaching dimension must be at least making that sort of worst-case analysis 
uninteresting. These differences present unique challenges to learning. 



2 Motivation and related work 

The problem of learning a ranking from few pairwise comparisons is motivated by what we perceive 
as a significant gap in the theory of ranking and permutation learning. Most work in ranking with 
structural constraints assumes a passive approach to learning; pairwise comparisons or partial rank- 
ings are collected in a random or non-adaptive fashion and then aggregated to obtain a full ranking 
(cf. lfT2l[T3l[T4l[T5l ). However, this may be quite inefficient in terms of the number of pairwise com- 
parisons or partial rankings needed to learn the (full) ranking. This inefficiency was recently noted 
in the related area of social choice theory [ 16 1. Furthermore, empirical evidence suggests that, even 
under complex ranking models, adaptively selecting pairwise comparisons can reduce the number 
needed to learn the ranking IfTTI . It is cause for concern since in many applications it is expen- 
sive and time-consuming to obtain pairwise comparisons. For example, psychologists and market 
researchers collect pairwise comparisons to gauge human preferences over a set of objects, for sci- 
entific understanding or product placement. The scope of these experiments is often very limited 
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simply due to the time and expense required to collect the data. This suggests the consideration of 
more selective and judicious approaches to gathering inputs for ranking. We are interested in taking 
advantage of underlying structure in the set of objects in order to choose more informative pairwise 
comparison queries. From a learning perspective, our work adds an active learning component to a 
problem domain that has primarily been treated from a passive learning mindset. 

We focus on pairwise comparison queries for two reasons. First, pairwise comparisons admit a 
halfspace representation in embedding spaces which allows for a geometrical approach to learning in 
such structured ranking spaces. Second, pairwise comparisons are the most common form of queries 
in many applications, especially those involving human subjects. For example, consider the problem 
of finding the most highly ranked object, as illustrated by the following familiar task. Suppose 
a patient needs a new pair of prescription eye lenses. Faced with literally millions of possible 
prescriptions, the doctor will present candidate prescriptions in a sequential fashion followed by 
the query: better or worse? Even if certain queries are repeated to account for possible inaccurate 
answers, the doctor can locate an accurate prescription with just a handful of queries. This is possible 
presumably because the doctor understands (at least intuitively) the intrinsic space of prescriptions 
and can efficiently search through it using only binary responses from the patient. 

We assume that the objects can be embedded in R d and that the distances between objects and 
the reference are consistent with the ranking (Assumption Al). The problem of learning a general 
function / : M. d —> M using just pairwise comparisons that correctly ranks the objects embedded in 
M. d has previously been studied in the passive setting lTT2l [13] [T4l [T31 . The main contributions of 
this paper are theoretical bounds for the specific case when f(x) = \\x — r CT || where r a <E K d is 
the reference point. This is a standard model used in multidimensional unfolding and psychometrics 
QjTj and one can show that this model also contains the familiar functions f(x) = r^x for 
all r„ € Mr. We are unaware of any existing query-complexity bounds for this problem. We do 
not assume a generative model is responsible for the relationship between rankings to embeddings, 
but one could. For example, the objects might have an embedding (in a feature space) and the 
ranking is generated by distances in this space. Or alternatively, structural constraints on the space 
of rankings could be used to generate a consistent embedding. Assumption Al, while arguably quite 
natural/reasonable in many situations, significantly constrains the set of possible rankings. 

3 Geometry of rankings from pairwise comparisons 

The embedding assumption Al gives rise to geometrical interpretations of the ranking problem, 
which are developed in this section. The pairwise comparison qij can be viewed as the membership 
query: is 9i ranked before 9j in the (full) ranking al The geometrical interpretation is that qi j re- 
quests whether the reference r a is closer to object 8 L or object 6j in R d . Consider the line connecting 
9i and 8j in E rf . The hyperplane that bisects this line and is orthogonal to it defines two halfspaces: 
one containing points closer to Oi and the other the points closer to 9j. Thus, qij is a membership 
query about which halfspace r CT is in, and there is an equivalence between each query, each pair of 
objects, and the corresponding bisecting hyperplane. The set of all possible pairwise comparison 
queries can be represented as (™) distinct halfspaces in M. d . The intersections of these halfspaces 
partition R d into a number of cells, and each one corresponds to a unique ranking of 0. Arbitrary 
rankings are not possible due to the embedding assumption Al, and recall that the set of rankings 
possible under Al is denoted by £ n ,d. The cardinality of E Jl £ ; is equal to the number of cells in the 
partition. We will refer to these cells as <i-cells (to indicate they are subsets in d-dimensional space) 
since at times we will also refer to lower dimensional cells; e.g., (d — l)-cells. 

3.1 Counting the number of possible rankings 

The following lemma determines the cardinality of the set of rankings, Y, n ,d, under assumption Al. 

Lemma 1. [7] Assume Al-2. Let Q(n, d) denote the number of d-cells defined by the hyperplane ar- 
rangement of pairwise comparisons between these objects (i.e. Q(n, d) — |S niC j|J. Q(n, d) satisfies 
the recursion 

Q(n, d) = Q(n - 1, d) + (n - l)Q(n - 1, d - 1) , where Q(l, d) = 1 and Q(n, 0) = 1. (3) 
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In the hyperplane arrangement induced by the n objects in d dimensions, each hyperplane is inter- 
sected by every other and is partitioned into Q(n — 1, d — 1) subsets or (d — l)-cells. The recursion, 
above, arises by considering the addition of one object at a time. Using this lemma in a straightfor- 
ward fashion, we prove the following corollary in Appendix |A.2| 

Corollary 1. Assume Al-2. There exist positive real numbers k\ and fc 2 such that 

n 2d n 2d 

for n > d + 1. If n < d + 1 then Q(n, d) = n\. For n sufficiently large, k\ ~ 1 and ki — 1 suffice. 



3.2 Lower bounds on query complexity 

Since the cardinality of the set of possible rankings is |£ n ,d| = Q(n,d), we have a simple lower 
bound on the number of queries needed to determine the ranking. 

Theorem 1. Assume Al-2. To reconstruct an arbitrary ranking a £ £ n ,d any algorithm will require 
at least log 2 |E n .<i| = 0(2dlog 2 n) pairwise comparisons. 

Proof. ByCorollary[l]|S, i:d | = Q{n 2d ), and so at least 2d log n bits are needed to specify a ranking. 
Each pairwise comparison provides at most one bit. □ 

If each query provides a full bit of information about the ranking, then we achieve this lower bound. 
For example, in the one-dimensional case (d = 1) the objects can be ordered and binary search 
can be used to select pairwise comparison queries, achieving the lower bound. This is generally 
impossible in higher dimensions. Even in two dimensions there are placements of the objects (still 
in general position) that produce <i-cells in the partition induced by queries that have n — 1 faces (i.e., 
bounded by n — 1 hyperplanes) as shown in Appendix A. 3 It follows that the worst case situation 
may require at least n — 1 queries in dimensions d > 2. In light of this, we conclude that worst case 
bounds may be overly pessimistic indications of the typical situation, and so we instead consider the 
average case performance introduced in Section fTTT] 



3.3 Inefficiency of random queries 

The geometrical representation of the ranking problem reveals that randomly choosing pairwise 
comparison queries is inefficient relative to the lower bound above. To see this, suppose m queries 
were chosen uniformly at random from the possible (™) . The answers to m queries narrows the 
set of possible rankings to a rf-cell in R d . This <i-cell may consist of one or more of the d-cells 
in the partition induced by all queries. If it contains more than one of the partition cells, then the 
underlying ranking is ambiguous. 

Theorem 2. Assume Al-2. Let N = Cl\ Suppose m pairwise comparison are chosen uniformly at 
random without replacement from the possible Then for all positive integers N > m > d the 
probability that the m queries yield a unique ranking is (™) /(^) < (^) d - 

Proof. No fewer than d hyperplanes bound each d-cell in the partition of M. d induced by all possible 
queries. The probability of selecting d specific queries in a random draw of rn is equal to 

/N-d\ /(N\ _ fm\ / (N\ < m d d d < / m\ d d d < / em\ d _ 
\m - d) I \m) ~ \d J / \d J ~ IftW- ~ \N J dT ~ \~W ) ' L 

Note that (™)/(^) < 1/2 unless m = Q{n 2 ). Therefore, if the queries are randomly chosen, then 
we will need to ask almost all queries to guarantee that the inferred ranking is probably correct. 



4 Analysis of sequential algorithm for query selection 

Now consider the basic sequential process of the algorithm in Figure [T] Suppose we have ranked 
k — 1 of the n objects. Call these objects 1 through k — 1. This places the reference r a within 
a d-cell (defined by the labels of the comparison queries between objects 1, . . . , k — 1). Call this 
d-cell Cfc_i. Now suppose we pick another object at random and call it object k. A comparison 
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query between object fc and one of objects 1, . . . , k — 1 can only be informative (i.e., ambiguous) 
if the associated hyperplane intersects this d-cell Ck-i (see Figure 2j. If k is significantly larger 
than d, then it turns out that the cell Ck-i is probably quite small and the probability that one of the 
queries intersects Ck-i is very small; in fact the probability is on the order of 1/fc 2 . 



4.1 Hyperplane-point duality 

Consider a hyperplane h = (ho, hi, . . . , hj) with (d + 1) parameters in M. d and a point p = 
(pi, . .. ,pd) £ K d that does not lie on the hyperplane. Checking which halfspace p falls in, i.e., 
hxpi + h 2 p2 + • • • + hdPd + ho ^ 0, has a dual interpretation: h is a point in R d+1 and p is a 
hyperplane in R d+1 passing through the origin (i.e., with d free parameters). 

Recall that each possible ranking can be represented by a reference point r a £ R d . Our problem is 
to determine the ranking, or equivalently the vector of responses to the (™) queries represented by 
hyperplanes in R d . Using the above observation, we see that our problem is equivalent to finding a 
labeling over (™) points in R d+1 with as few queries as possible. We will refer to this alternative 
representation as the dual and the former as the primal. 



4.2 Characterization of an ambiguous query 

The characterization of an ambiguous query has interpretations in both the primal and dual spaces. 
We will now describe the interpretation in the dual which will be critical to our analysis of the 
sequential algorithm of Figure [T] 

Definition 3. 421/ Let S be a finite subset ofW 1 and let S + C S be points labeled +1 and S~ — 
S \ S + be the points labeled — 1 and let x be any other point except the origin. If there exists two 
homogeneous linear separators of S + and S~ that assign different labels to the point x, then the 
label ofx is said to be ambiguous with respect to S. 

Lemma 2. [8, Lemma 1 ] The label of x is ambiguous with respect to S if and only if S + and S~ 
are homogeneously linearly separable by a (d — I) -dimensional subspace containing x. 

Let us consider the implications of this lemma to our scenario. Assume that we have labels for all the 
pairwise comparisons of k — 1 objects. Next consider a new object called object k. In the dual, the 
pairwise comparison between object k and object i, for some i £ {1, . . . , fc — 1}, is ambiguous if and 
only if there exists a hyperplane that still separates the original points and also passes through this 
new point. In the primal, this separating hyperplane corresponds to a point lying on the hyperplane 
defined by the associated pairwise comparison. 



4.3 The probability that a query is ambiguous 

An essential component of the sequential algorithm of Figure [T] is the initial random order of the 
objects; every sequence in which it could consider objects is equally probable. This allows us to 
state a nontrivial fact about the partial rankings of the first k objects observed in this sequence. 

Lemma 3. Assume Al-2 and a ~ U. Consider the subset S C with \S\ = k that is randomly 
selected from such that all (?) subsets are equally probable. IfEk,d denotes the set of possible 
rankings of these k objects then every a £ ^k,d ' s equally probable. 

Proof. Let a fc-partition denote the partition of R d into Q(k,d) d-cells induced by k objects for 
1 < k < n. In the n-partition, each d-cell is weighted uniformly and is equal to 1/Q(n, d). If we 
uniformly at random select k objects from the possible n and consider the fc-partition, each d-cell in 
the fc-partition will contain one or more o?-cells of the n-partition. If we select one of these d-cells 
from the fc-partition, on average there will be Q(n, d) /Q(k, d) d-cells from the n-partition contained 
in this cell. Therefore the probability mass in each d-cell of the fc-partition is equal to the number 
of cells from the n-partition in this cell multiplied by the probability of each of those cells from the 
n-partition: Q(n,d)/Q(k,d) x l/Q(n,d) = 1/Q(k, d), and = Q(k,d). □ 



As described above, for 1 < i < k some of the pairwise comparisons g^fc+i may be ambiguous. 
The algorithm chooses a random sequence of the n objects in its initialization and does not use 
the labels of qi k+i> ■ ■ ■ , Qj-i,k+i,Qj+i,k+ii ■ ■ ■ > Qk,k+i to make a determination of whether or not 
qj,k+i is ambiguous. It follows that the events of requesting the label of q^k+i f° r * — 1, 2, . . . , fc 
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are independent and identically distributed (conditionally on the results of queries from previous 
steps). Therefore it makes sense to talk about the probability of requesting any one of them. 

Lemma 4. Assume Al-2 and a ~ hi. Let A(k, d,lA) denote the probability of the event that the 
pairwise comparison qt t k+i is ambiguous for i — 1,2, ... ,k. Then there exists a positive, real 
number constant a independent ofk such that for k > 2d, A(k, d,U) < a|f. 

Proof. By Lemma 2] a point in the dual (pairwise comparison) is ambiguous if and only if there 
exists a separating hyperplane that passes through this point. This implies that the hyperplane rep- 
resentation of the pairwise comparison in the primal intersects the cell containing r a (see Figure [2] 
for an illustration of this concept). Consider the partition of R d generated by the hyperplanes cor- 
responding to pairwise comparisons between objects 1, . . . , k. Let P(k, d) denote the number of 
d-cells in this partition that are intersected by a hyperplane corresponding to one of the queries 

Qi,k+i> i £ {1, . . . , k}. Then it is not difficult to show that P(k, d) is bounded above by a con- 

,2(d-i) i r~| 

stant independent of n and k times 2 d - 1 (d-iy. ( see Appendix A.4i. By Lemma 3 every a-cell 



the partition induced by the k objects corresponds to an equally probable ranking of those objects. 
Therefore, the probability that a query is ambiguous is the number of cells intersected by the corre- 
sponding hyperplane divided by the total number of d-cells, and therefore A(k, d, U) — q^'^ • The 
result follows immediately from the bounds on P(k, d) and Corollary[T] □ 

Because the individual events of requesting each query are conditionally independent, the total num- 
ber of queries requested by the algorithm is just M n = 53fc=i Si=i l{Request g^fc+i}. Using the 



results above, it straightforward to prove the main theorem below (see Appendix | A. 5) 

Theorem 3. Assume Al-2 and a ~ U. Let the random variable M n denote the number of pairwise 
comparisons that are requested in the algorithm of Figure^ then 

^u[M n ] < \2da\ \og 2 n. 

Furthermore, if a ~ 7r and max a ^ n d ir a < c\T, n ^\ for some c > 0, then E 7r [Af„] < cE^[A/„]. 



5 Robust sequential algorithm for query selection 

We now extend the algorithm of Figure [T] to situations in which the response to each query is only 
probably correct. If the correct label of a query qi j is yij, we denote the possibly incorrect response 
by Yij. Let the probability that Yij = yij be equal to 1 —p, p < 1/2. The robust algorithm operates 
in the same fashion as the algorithm in Figure [T] with the exception that when an ambiguous query 
is encountered several (equivalent) queries are made and a decision is based on the majority vote. 
We will now judge performance based on two metrics: (i) how many queries are requested and (ii) 
how accurate the estimated ranking is with respect to the true ranking before it was corrupted. For 
any two rankings a, a we adopt the popular Kendell-Tau distance lfl9ll 

d T (a,a)= iMfi W 

(i,j):o-(i)<o-(j) 

where 1 is the indicator function. Clearly, d T (a,a) — d T (a,a) and < d T (a,cr) < Q). For any 
ranking a e ^ n ,d we wish t° find an estimate a G that is close in terms of d T (a, a) without 
requesting too many pairwise comparisons. For convenience, we will some times report results in 
terms of the proportion e of incorrect pairwise orderings such that d T (a,a) < ^(n)- Using the 
equivalence of the Kendell-Tau and Spearman's footrule distances (see [20|), if d T (a,a) < e(") 
then each object in u is, on average, no more than 0(en) positions away from its position in a. 
Thus, the Kendell-Tau distance is an intuitive measure of closeness between two rankings. 

First consider the case in which each query can be repeated to obtain multiple independent responses 
(votes) for each comparison query. This random errors model arises, for example, in social choice 
theory where the "reference" is a group of people, each casting a vote. The elementary proof of the 



next theorem is given in Appendix A. 6 
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Theorem 4. Assume Al-2 and a ~ U but that each response to the query qt t j is a realization of 
an i.i.d. Bernoulli random variable Yij with P{Yi,j ^ Ui.j) < P < 1/2 for all distinct i,j € 
{1, . . . , n}. If all ambiguous queries are decided by the majority vote of R independent responses 
to each such query, then with probability greater than 1 — 2nlog 2 (n) exp(— i(l — 2p) 2 R) this 
procedure correctly identifies the correct ranking (i.e. e = 0) and requests no more than 0(Rd log n) 
queries on average. 

We can deduce from the above theorem that to exactly recover the true ranking under the stated 
conditions with probability 1 — 6, one need only request 0(<i(l — 2p)~ 2 log 2 (n/5)) pairwise com- 
parisons, on average. 

In other situations, if we ask the same query multiple times we may get the same, possibly incorrect, 
response each time. This persistent errors model is natural, for example, if the reference is a single 
human. Under this model, if two rankings differ by only a single pairwise comparison, then they 
cannot be distinguished with probability greater than 1 — p. So, in general, exact recovery of the 
ranking cannot be guaranteed with high probability. The best we can hope for is to exactly recover 
a partial ranking of the objects (i.e. the ranking over a subset of the objects) or a ranking that is 
merely probably approximately correct in terms of the Kendell-Tau distance of Q. We will first 
consider the task of exact recovery of a partial ranking of objects and then turn our attention to the 
recovery of an approximate ranking. Henceforth, we will assume the errors are persistent. 



5.1 Robust sequential algorithm for persistent errors 

The robust query selection algorithm for persistent errors is presented in Figure|3] The key ingredient 
in the persistent errors setting is the design of a voting set for each ambiguous query encountered. 
Suppose the query j is ambiguous in the algorithm of Figure [T] In principle, a voting set could 
be constructed using objects ranked between i and j. If object k is between i and j, then note that 
Ui.j = Ui.k = Uk,j- In practice, we cannot identify the subset of objects ranked between i and j 
exactly, but we can find a set that contains them. For an ambiguous query q^ j define 

Tij :— {k € {1 . . . , ft } : g,^, q^j, or both are ambiguous}. (5) 

Then T^j contains all objects ranked between i and j (if k is ranked between i and j, and qi^ and 
qk j are unambiguous, then so is q^j , a contradiction). Furthermore, if the first j — 1 objects ranked in 
the algorithm were selected uniformly at random (or initialized in a random order in the algorithm) 
Lemma [3] implies that each object in Tjj is ranked between i and j with probability at least 1/3 



due to the uniform distribution over the rankings E Tl c ; (see Appendix A. 7 for an explanation). Tij 
will be our voting set. If we follow the sequential procedure of the algorithm of Figure [3] the 
first query encountered, call it q\ t %, will be ambiguous and T% t 2 will contain all the other n — 2 
objects. However, at some point for some query q.- L ,j it will become probable that the objects i and 
j are closely ranked. In that case, T^j may be rather small, and so it is not always possible to 
find a sufficiently large voting set to accurately determine yij. Therefore, we must specify a size- 
threshold R > 0. If the size of 7$ j is at least R, then we draw R indices from Tij uniformly at 
random without replacement, call this set {ti}fL 1 , and decide the label for qij by voting over the 
responses to {qi } k,qk,j '■ k € {^}/Li}; otherwise we pass over object j and move on to the next 
object in the list. Given that \Tij\ > R the label of qij is determined by: 

| l{^,fc = lAF feJ = l}-l{F i)fe =0An J =0}. (6) 

In the next section we will analyze this algorithm and show that it enjoys a very favorable query 
complexity while also admitting a probably approximately correct ranking. 



5.2 Analysis of the robust sequential algorithm 

Consider the robust algorithm in Figure [3] At the end of the process, some objects that were passed 
over may then be unambiguously ranked (based on queries made after they were passed over) or 
they can be ranked without voting (and without guarantees). As mentioned in Section 5.1 if the first 
j — 1 objects ranked in the algorithm of Figure[3]were chosen uniformly at random from the full set 
(i.e., none of the first j — 1 objects were passed over) then there is at least a one in three chance each 
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Robust Query Selection Algorithm 



, 9 n } in uniformly random order, 



input: n objects in M. d , R > 
initialize: objects = {9i,. 
6' = 6 

for j=2,. . . ,n 
fori=l,...,j-l 

if qij is ambiguous, 
Ti j ; := {k G {1 . . . , n} : qi k,qk j, or both are ambiguous} 
tt\Tij\>R 

{U}f =1 ' - uniform; /, 
request Y ijk , Y kj for all k <E {U}^ =1 
decide label of qij with (JgJ) 
else 

e 3 , + i 

else 

impute qij's label from previously labeled queries, 
output: ranking over objects in 9' 



Figure 3: Robust sequential algorithm for selecting queries of Sec- 
5.1 See Figure [2] and Section 4.2 for the definition of an am- 



tion 



biguous query. 

object in Tij for some ambiguous query 4 is r anked between i and j. With this in mind, we have 
the following theorem, proved in Appendix |A.7| 

Theorem 5. Assume Al-2, a — U, and P(Yi t j 7^ Ui t j) — p. For every set Tij constructed in 
the algorithm of Figure^ assume that an object selected uniformly at random from Tjj is ranked 
between 9i and 9j with probability at least 1/3. Then for any size-threshold R > 1, with probability 
greater than 1 — 2n log 2 (n) exp ( — |(1 — 2p) 2 i?) the algorithm correctly ranks at least n/(2R+l) 
objects and requests no more than 0(Rd\ogn) queries on average. 

Note that before the algorithm skips over an object for the first time, all objects that are ranked at such 
an intermediate stage are a subset chosen uniformly at random from the full set of objects, due to 
the initial randomization. Therefore, if Tij is a voting set in this stage, an object selected uniformly 
at random from Tij is ranked between 9i and 9j with probability at least 1/3, per Lemma [3] After 
one or more objects are passed over, however, the distribution is no longer necessarily uniform 
due to this action, and so the assumption of the theorem above may not hold. The procedure of 
the algorithm is still reasonable, but it is difficult to give guarantees on performance without the 
assumption. Nevertheless, this discussion leads us to wonder how many objects the algorithm will 



rank before it skips over its first object. The next lemma is proved in Appendix A. 8 

Lemma 5. Consider a ranking of n objects and suppose objects are drawn sequentially, cho 
sen uniformly at random without replacement. If M is the largest integer such that M ob 
jects are drawn before any object is within R positions of another one in the ranking, 

M > 



then 



n/R 
61og(2) 



with probability at least 



61og(2) 



( e -(V 61 °g( 2 ) fl /™+ 1 ) 2 /2 . 



-n/(SR)\Asn/R- 



P(M > 



n/R 
61og(2) 



Uog(2)' 



Lemma|5]characterizes how many objects the robust algorithm will rank before it passes over its first 
object because if there are at least R objects between every pair of the first M objects, then Tj j > R 
for all distinct i,j 6 {1, . . . , M} and none of the first M objects will be passed over. We can 
conclude from Lemma|5]and Theorem|5]that with constant probability (with respect to the initial or- 
dering of the objects and the randomness of the voting), the algorithm of Figure[3]exactly recovers a 
partial ranking of at least f2(y(I — 2p) 2 n/logn) objects by requesting just 0{d{\ — 2p)" 2 log 2 n) 
pairwise comparisons, on average, with respect to all the rankings in Yj n ^. If we repeat the algorithm 
with different initializations of the objects each time, we can boost this constant probability to an 
arbitrarily high probability (recall that the responses to queries will not change over the repetitions). 
Note, however, that the correctness of the partial ranking does not indicate how approximately cor- 
rect the remaining rankings will be. If the algorithm of Figure [3] ranks m objects before skipping 
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21og 2 |£„,«j| 




Dimension 

Figure 4: Mean and standard deviation of re- 
quested queries (solid) in the error-free case for 
n = 100; log 2 |X n ,<j| is a lower bound (dashed). 



Table 1 : Statistics for the algorithm robust to 
persistent errors of Section [5] with respect to 
all (™) pairwise comparisons. Recall y is the 
noisy response vector, y is the embedding's 
solution, and y is the output of the robust al- 
gorithm. 



Dimension 


2 


3 


% of queries 
requested 


mean 


14.5 


18.5 


std 


5.3 


6 


Average error 


d{y,y) 


0.23 


0.21 


d{y,y) 


0.31 


0.29 



over its first, then the next lemma quantifies how accurate an estimated ranking is in terms of Kendel- 
Tau distance, given that it is some ranking in E„.d that is consistent with the probably correct partial 
ranking of the first m objects (the output ranking of the algorithm may contain more than m objects 



but we make no guarantees about these additional objects). The proof is available in Appendix A. 9 

Lemma 6. Assume Al-2 and a ~ U. Suppose we select 1 < m < n objects uniformly at random 
from the n and correctly rank them amongst themselves. If a is any ranking in T, n-C i that is consistent 
with all the known pairwise comparisons between the m objects, then E[d r (<7, a)] — 0(d/m 2 ) (™), 
where the expectation is with respect to the random selection of objects and the distribution of the 
rankings li. 

Combining Lemmas [5] and [6] in a straightforward way, we have the following theorem. 

Theorem 6. Assume Al-2, a ~ U, and P(¥i,j ^ Vi,j) = P- If R — 0((1 — 2p)~ 2 logn) and a 
is any ranking in S„ !( i that is consistent with all known pairwise comparisons between the sub- 
set of objects ranked in the output of the algorithm of Figure [3] then with constant probability 
E[d T (<7, a)] — 0(d(l — 2p)~ 2 log(n)/n) Q) and no more than 0(d(l — 2p)~ 2 log 2 (n)) pairwise 
comparisons are requested, on average. 

If we repeat the algorithm with different initializations of the objects until a sufficient number of 
objects are ranked before an object is passed over, we can boost this constant probability to an 
arbitrarily high probability. However, in practice, we recommend running the algorithm just once to 
completion since we do not believe passing over an object early on greatly affects performance. 



6 Empirical results 

In this section we present empirical results for both the error-free algorithm of Figure [T] and the 
robust algorithm of Figure [3] For the error-free algorithm, n — 100 points, representing the 
objects to be ranked, were uniformly at random simulated from the unit hypercube [0,1] d for 
d = 1, 10, 20, . . . , 100. The reference was simulated from the same distribution. For each value 
of d the experiment was repeated 25 times using a new simulation of points and the reference. Be- 
cause responses are error-free, exact identification of the ranking is guaranteed. The number of 
requested queries is plotted in Figure [4] with the lower bound of Theorem [T] for reference. The 
number of requested queries never exceeds twice the lower bound which agrees with the result of 
Theorem [3] 

The robust algorithm of Figure[3]was evaluated using a symmetric similarity matrix dataset available 
at ETI whose (i,j)th entry, denoted Sjj, represents the human-judged similarity between audio 
signals i and j for all i ^ j 6 { 1 , . . . , 100}. If we consider the fcth row of this matrix, we can rank 

the other signals with respect to their similarity to the fcth signal; we define qij :— {si^i > Sk,j} 

and y^ 1 ) := }. Since the similarities were derived from human subjects, the derived labels 

may be erroneous. Moreover, there is no possibility of repeating queries here and so the errors are 
persistent. The analysis of this dataset in |2 1 suggests that the relationship between signals can be 
well approximated by an embedding in 2 or 3 dimensions. We used non-metric multidimensional 
scaling [5] to find an embedding of the signals: 6\, . . . , 6*ioo G K d for d = 2 and 3. For each object 
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0k, we use the embedding to derive pairwise comparison labels between all other objects as follows: 
y^j := 1{ | \9k — 0i\ | < 1 1 Ok — 0j\\}, which can be considered as the best approximation to the labels 

(k) 

y\ ! (defined above) in this embedding. The output of the robust sequential algorithm, which uses 

only a small fraction of the similarities, is denoted by yf^ ■ We set R — 15 using Theorem^ as a 

rough guide. Using the popular Kendell-Tau distance d(y^ k \y ( - k " 1 ) = Q) 1 2~2i<j ^-{Vif ^ vfj} 
|[T9l for each object k, we denote the average of this metric over all objects by d(y, y) and report 
this statistic and the number of queries requested in Table [T] Because the average error of y is only 
0.07 higher than that of y, this suggests that the algorithm is doing almost as well as we could hope. 
Also, note that 2R2d\ogn/ is equal to 11.4% and 17.1% for d = 2 and 3, respectively, which 
agrees well with the experimental values. 
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A Appendix 

A.l Computational complexity and implementation 

The computational complexity of the algorithm in Figure [T] is determined by the complexity of 
testing whether a query is ambiguous or not and how many times we make this test. As written in 
Figure [l] the test would be performed 0(n 2 ) times. But if binary sort is used instead of the brute- 
force linear search this can be reduced to n log 2 n and, in fact, this is implemented in our simulations 
and the proofs of the main results. The complexity of each test is polynomial in the number of 
queries requested because each one is a linear constraint. Because our results show that no more than 
0(d\ogn) queries are requested, the overall complexity is no greater than 0(n poly (d) poly (log n)). 

A.2 Proof of Corollary [j] 

Proof. For initial conditions given in Lemma[T] if d <C n — 1 a simple manipulation of ([3]l shows 

n-1 

Q(n,d) = l + ^2(n-i)Q(n-i,d-l) 

i=i 
n-l 



l + 5^<Q(*,d- 1) 

i=i 

71—1 r i— 1 

i + i + £iQCj\<*-2) 



i=l 3 = 1 

n—1 i—1 r i—1 
l + e(n 2 /2) + ]T^zj l + ]TfcQ(M-3) 

i=l j=l L k=l 

n—1 i—1 j—1 i- 

1 + 6(n 2 /2) + 9(n 4 /2/4) + £ £ £ ij k 1 + £ I Q(l, d - 4) 

i=l j=l k=l 



fc-1 



Z=l 



1 + 6(n72) + • • • + 



\2 d d\ ' 



From simulations, this is very tight for large values of n. If d > n — 1 then Q(n, d) = nl because 
any permutation of n objects can be embedded in n — 1 dimensional space [7 1. □ 



A.3 Construction of a c?-cell with n—1 sides 

Situations may arise in which Sl(ri) queries must be requested to identify a ranking because the 
d-cell representing the ranking is bounded by n — 1 hyperplanes (queries) and if they are not all 
requested, the ranking is ambiguous. We now show how to construct this pathological situation in 
R 2 . Let 9 be a collection of n points in R 2 where each fie 6 satisfies Q\ = 6 2 and Q x € [0, 1] 
where Oi denotes the zth dimension of 9 (i G {1, 2}). Then there exists a 2-cell in the hyperplane 
arrangement induced by the queries that has n — 1 sides. This follows because the slope of the 
parabola keeps increasing with 9\ making at least one query associated with (n — 1) 6>'s bisect the 
lower-left, unbounded 2-cell. This can be observed in Figure [5] Obviously, a similar arrangement 
could be constructed for all d > 2. 



12 




Figure 5: The points 9 representing the objects are dots on the right, the lines are the queries, and 
the black, bold lines are the queries bounding the n — 1 sided 2-cell. 



A.4 Proof of Lemma H 

Proof. Here we prove an upper bound on P(k, d). P(k, d) is equal to the number of tf-cells in the 
partition induced by objects 1, . . . , k that are intersected by a hyperplane corresponding to a pair- 
wise comparison query between object k + 1 and object i, i E {1, . . . , k}. This new hyperplane is 
intersected by all the („) hyperplanes in the partition. These intersections partition the new hyper- 
plane into a number of (d — l)-cells. Because the (k + l)st object is in general position with respect 
to objects 1, . . . , k, the intersecting hyperplanes will not intersect the hyperplane in any special or 
non-general way. That is to say, the number of (d — 1) -cells this hyperplane is partitioned into is 
the same number that would occur if the hyperplane were intersected by ( 2 ) hyperplanes in general 
position. Let K — (*) for ease of notation. It follows then from [8, Theorem 3] that 



i=0 v 7 i=0 v 7 k=0 



2H\J \2 d - l {d-l)\ 



□ 



A.5 Proof of Theorem^ 



Proof. Let B^+i denote the total number of pairwise comparisons requested of the (k + l)st object; 
i.e., number of ambiguous queries in the set q^k+u * = 1, ■ ■ ■ ,k. Because the individual events 
of requesting these are conditionally independent (see Section |4.3[ l, it follows that each B^+i is 
an independent binomial random variable with parameters A(k, d,U) and k. The total number of 
queries requested by the algorithm is 



n— 1 k 



M n =^2J2 H^uest q itk+1 } = ^ B k+1 . (7) 



fc=l i=l fe=l 



Because Lemma [4] is only relevant for sufficiently large k, we assume that none of the pairwise 



comparisons are ambiguous when k < 2da. Recall from Section A.l that binary sort is implemented 
so for these first \2da] objects, at most \2da] log 2 ([~2da] ) queries are requested. For k > 2da the 
number of requested queries to the fcth object is upper bounded by the number of ambiguous queries 
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of the kth object. Then using the known mean and variance formulas for the binomial distribution 



n-l 

E u [M n ] = ^E w [£? fc+1 ] 

\2da] n-l , 

E„ x -> Ada 

Bk+1 + E x 

fe=2 fc=T2da]+l 

< [2da] log 2 |~2da| + 2ctalog (n/[2da]) 

< [~2da] log 2 n 



We now consider the case for a general distribution n. Enumerate the rankings of S„ )t j. Let iVj 
denote the (random) number of requested queries needed by the algorithm to reconstruct the ith 
ranking. Note that the randomness of iV, is only due to the randomization of the algorithm. Let m 
denote the probability it assigns to the ith ranking as in Definition [T] Then 

Q(n,d) 

Assume that the distribution over rankings is bounded above such that no ranking is overwhelmingly 
probable. Specifically, assume that the probability of any one ranking is upper bounded by c/ Q(n, d) 
for some constant c > 1 that is independent of n. Under this bounded distribution assumption, 
E7r[M n ] is maximized by placing probability c/Q(n,d) on the k := Q(n,d)/c cells for which 
K[Ni] is largest (we will assume k is an integer, but it is straightforward to extend the following 
argument to the general case). Since the mass on these cells is equal, without loss of generality 
we may assume that E[A^] = p, a common value on each, and we have E„.[M„] = \l. For the 
remaining Q(n,d) — k cells we know that K[Ni] > d, since each cell is bounded by at least d 
hyperplanes/queries. Under these conditions, we can relate E ff [A/„] to E^[M„] as follows. First 
observe that 

Q{n,d) f-J Q(n,dj Q(n,d) 

which implies 

E n [M n ] = M < 9^(E u [M n }-d^^-) - c(E u [M n ]-d^^) < cE u [M n ] . 

In words, the non-uniformity constant c > 1 scales the expected number of queries. Under Al-2, 
for large n we have E^ [M n ] = 0(c d log n). □ 



A.6 Proof of Theorem [4] 

Proof. Suppose q^j is ambiguous. Let a be the frequency of Yij = 1 after R trials. Let E[d] = a. 
The majority vote decision is correct if \a — a\ < 1/2 — p. By Chernoff's bound, P(|a — a\ > 
1/2 — p) < 2 exp(— 2(1/2 — p) 2 R)- The result follows from the union bound over the total number 



of queries considered: n log 2 n (See Appendix A. 1 1 



A.7 Proof of Theoremg] 

Suppose Qij is ambiguous. Let Sij denote the subset of such that 6k € if it is ranked 
between objects 0, and 0j (i.e. Sij = {9 k € 6 : Qi -< k -< 0j or 9j -< 9 k ~< 6i}). Note that 
Vi,j = Vi.k = Vk,j if and only if 6 k G S id . If we define = l{Y itk = 1 A Y k j = 1} - l{Y^ k = 
A Yfe j = 0}, where 1 is the indicator function, then for any subset T C 8 such that Sij C T, 
the sign of the sum X^ fc eT ^ij ^ s a predictor of j/jj. In fact, with respect to just the random errors, 
E[| ^2g k<ET Efj |] = — 2p). To see this, without loss of generality let yij — 1, then for 
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8 k £ Si j 



K[E^] = E[l{Y hk = 1 A Y ktj = 1} - l{Y hk = A Y kjj = 0}] 

= P(y iifc = 1 A Y kJ = 1) - P(Y a = A Y kd = 0) 

= (l-pf-p 2 
= l-2p. 

If 0fc ^ Si j then it can be shown by a similar calculation that E = 0. 

To identify Si.j we use the fact that if 9 k £ Sij then qi ik , qj tk , or both are also ambiguous simply 
because otherwise qij would not have been ambiguous in the first place (Figure[6]may be a useful 
aid to see this). While the converse is false, Lemma [3] says that each of the six possible rankings 
of {9i, 9j,9k] are equally probable if they were uniformly at random chosen (thus partly justifying 
this explicit assumption in the theorem statement). It follows that if we define the subset Tjj <E to 
be those objects 9 k with the property that qi <k , q k j, or both are ambiguous then the probability that 
k £ Si.j is at least 1/3 if 9 k C Tjj. You can convince yourself of this using Figure [6] Moreover, 
E [\ EfeeT, - E i,j\] ^ I 7 * J K 1 ~ 2 P) / 3 which implies the sign of the sum ^2g k ^. . E ij i s a reliable 
predictor of q t f, just how reliable depends only on the size of T t j. 




Figure 6: Let q^j be ambiguous. Object k will be informative to the majority vote of jjij if the 
reference lies in the shaded region. There are six possible rankings and if q i k , q k j, or both are 
ambiguous then the probability that the reference is in the shaded region is at least 1/3 

Fix R > 0. Suppose qij is ambiguous and assume without loss of generality that yij = 1. Given 
that E[ J^keT- E ij] — i (1 — 2p)/3 from above, it follows from Hoeffding's inequality that 
the probability that J2keT, E tj < is less than exp ( - | (1 - 2p)' 2 \T it j |) . If only a subset of T id 
of size R is used in the sum then \T^j | is replaced by R in the exponent. This test is only performed 
when \Ti j\ > R and clearly no more times than the number of queries considered to rank n objects 
in the full ranking: nlogn n. Thus, all decisions using this test are correct with probability at least 
1 — 2nlog 2 (n) exp ( — ^(1 — 2p) 2 R). Only a subset of the n objects will be ranked and of those, 
2R + 1 times more queries will be requested than in the error-free case (two queries per object in 
Tij). Thus the robust algorithm will request no more than 0(Rd\og n) queries on average. 

To determine the number of objects that are in the partial ranking, let Q' C denote the subset of 
objects that are ranked in the output partial ranking. Each 9 k £ 0' is associated with an index in 
the true full ranking and is denoted by a(6 k ). That is, if a(9 k ) — 5 then it is ranked fifth in the full 
ranking but in the partial ranking could be ranked first, second, third, fourth, or fifth. Now imagine 
the real line with tick marks only at the integers 1, . . . , n. For each 9 k £ 0' place an i?-ball around 
each 9 k on these tick marks such that if <r(9k) = 5 and R = 3 then 2, . . . , 8 are covered by the ball 
around cr(9 k ) and 1 and 9, . . . , n are not. Then the union of the balls centered at the objects in 0' 
cover 1, . . . , n. If this were not true then there would be an object 9j </ 0' with \Si j | > R for all 
9i £ 0'. But Si j C Tij implies \Tij\ > R which implies j £ 0', a contradiction. Because at 
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least nj (2R + 1) i?-balls are required to cover 1, . . . , n, at least this many objects are contained in 
9'. ~ □ 



A.8 Proof of Lemma H] 



Proof. Assume M < If p m denotes the probability that the (to + l)st object is within R 
positions of one of the first to objects, given that none of the first to objects are within R positions 
of each other, then ^ < p m < and 



P(M = m)> Yl ( 1 



2RI \ Rm 



Taking the log we find 



log P{M = to) > log 

> log 

> log 

> log 

> log 



Rm 

n 

Rm 

n 

Rm 

n 
Rm 

n 
Rm 

n 



!=1 ^ 
TO — 1) log 



2RI 



m—1 



(m 



1) log \1 
l)log( 1 

1 



Rm 



1 - 



2RI 



n — m 
3Rm\ 



2n J 
31og(2)i?TO 



where the second line follows from Jensen's inequality, the fourth line follows from the fact that 
m < ih> an d the l ast li ne follows from the fact that (1 — x) > exp(— 2 \og(2)x) for x < 1/2. We 



conclude that P(M = to) > ^TOexp{-31og(2)^TO 2 }. Now if a = 

n/(3R)-l 

P(M>a)> V -TOexp{-31og(2)-m 2 } 
z — ' n n 



n/R 
61og(2) 



we have 



> 



i/(3R) R 



R 



a+l 
1 



xexp{— 31og(2)— x }dx 
n n 



-(^6 log(2)fl/n+l) 2 /2 _ -log(2)n/(3fl.) 



61og(2) 

where the second line follows from the fact that xe~ ax / 2 is monotonically decreasing for x > 
y/l/a. Note, P(M > & ^ 2) ) is greater than ^ for n/R > 7, and ^ for n/R > 40. Moreover, 

asn/i?->oo, P(M>J^k)^ . □ 



6v^log(2)- 
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A.9 Proof of Lemma 

Proof. Enumerate the objects such that the first m are the objects ranked amongst themselves. Let 
y be the pairwise comparison label vector for a and y be the corresponding vector for a. Then 

m k—1 n k—1 

k=2 1=1 k=m+l 1=1 

n k—1 

k=m+l 1=1 
n k — 1 

< Y Y P { ReqUeStqi > k \ labelSt ° qs< i m ^ m } 
k=m+l 1=1 

n k—1 „ j 

E v Zad 

k=m+l 1=1 

2ad (n — m)(n + m + 1) 

< ad V „ ; - 1 



where the third line assumes that every pairwise comparison that is ambiguous (that is, cannot be 
imputed using the knowledge gained from the first m objects) is incorrect. The fourth line follows 
from the application of Lemma[3]and Lemma|4] □ 
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